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Executive  Summary 


A  well-managed  organization  sets  goals  that  address  a  long-term  strategy.  The  organization 
defines  objectives  or  subgoals  that  satisfy  the  strategy,  creating  a  plan  and  applying  resources 
to  achieve  the  objectives.  The  organization  uses  data  that  track  the  state  of  the  effort  to 
determine  whether  the  goals  are  being  achieved  as  time  passes  (that  is,  whether  the  plan  is 
working).  By  tracking  and  analyzing  relevant  measurable  attributes  of  the  effort’s  process  and 
product  as  metrics,  the  organization’s  managers  track  progress  toward  achieving  the  goals. 
The  managers  can  also  detect  issues  that  indicate  when  the  effort  has  diverged  from 
expectations.  They  can  then  revise  the  goals,  plan,  or  resources  to  address  these  issues — or 
recalibrate  everyone’s  expectations  [Clements  02]. 

To  guide  decision  making  for  the  Naval  Undersea  Warfare  Center  (NUWC)  Division 
Newport  Ranges,  Engineering,  and  Analysis  Department’s  product  line  effort  [Cohen  02], 
management  has  created  a  measurement  program  for  its  product  line  work.  The  Goal-Driven 
Measurement  approach  [Park  96]  provided  the  basis  for  setting  up  the  program,  and  NUWC 
established  a  software  measurement  team  to  develop  and  monitor  the  program.  Data  from  the 
East  Coast  Shallow  Water  Test  Range  project  served  as  the  first  trial  for  the  NUWC 
measurement  program.  Four  other  projects  also  established  tracking  efforts.  The  approach 
will  be  built  into  NUWC’s  current  software  change  proposal  (SCP)  system  as  part  of  its 
product  line  operations. 

This  report  summarizes  the  steps  NUWC  has  taken  to  establish  a  measurement  program. 
NUWC  created  a  software  measurement  team  consisting  of  representatives  from  projects 
participating  in  the  product  line  effort.  That  team  established  goals  that  the  product  line 
effort  should  attain  and  formalized  questions  to  ask  about  the  effort  to  track  progress  toward 
those  goals.  The  team  also  identified  indicators  that  graphically  illustrate  progress  and  the 
data  elements  needed  to  produce  them.  The  report  concludes  with  a  summary  of  the  results 
to  date  and  a  list  of  the  next  steps  to  be  taken  to  refine  the  measurement  program. 
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Abstract 


The  Naval  Undersea  Warfare  Center  (NUWC)  Division  Newport  Ranges,  Engineering,  and 
Analysis  Department  applied  product  line  practices  in  the  development  of  systems  for  the 
U.S.  Navy  test  ranges  it  supports.  To  gauge  the  success  of  its  product  line  effort,  NUWC  must 
be  able  to  measure  the  effectiveness  of  the  product  line  approach  compared  to  more 
traditional  development  approaches.  To  do  this,  NUWC  established  a  software  measurement 
team  to  develop  and  monitor  a  measurement  program.  The  team  includes  representatives 
from  programs  using  the  NUWC  core  asset  base,  RangeWare.  Four  projects  are  currently 
contributing  to  the  measurement  program.  This  report  documents  NUWC’s  approach  for 
measurement  by  describing  the  Goal-Driven  Software  Measurement  approach  adopted  by  the 
test  range  product  line  effort  and  by  providing  early  results  of  the  measurement  program. 
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1  Introduction 


The  Naval  Undersea  Warfare  Center  (NUWC)1  Division  Newport  Ranges,  Engineering,  and 
Analysis  Department  instituted  a  measurement  program  to  track  progress  within  its  software 
product  line  activity.  The  RangeWare  core  asset  base  developed  at  NUWC  represents  the 
foundation  of  the  software  product  line  for  U.S.  Navy  range  systems  [Cohen  02].  The  product 
line  includes  approximately  seven  systems  already  installed,  with  five  to  six  new  projects  per 
year  that  support  range  systems  for  the  Navy.  The  measurement  program,  once  fully 
institutionalized,  will  look  at  the  entire  product  line  production  process  [Clements  02], 
including  core  asset  development,  product  development,  and  management  activities.  This 
report  describes  the  steps  NUWC  took  to  develop  the  measurement  program  and  to  establish 
a  measurement  team  with  representatives  from  the  various  product  line  projects. 

The  systems  in  NUWC’s  product  line  support  test  and  evaluation  (T&E)  and  training  ranges, 
encompass  a  number  of  mission  areas  including  live  open  air  ranges,  hardware-in-the-loop 
simulations,  and  other  specialized  simulations.  Software  capabilities  supported  by  the 
RangeWare  core  asset  base  include  resources  for  major  range  subsystems  such  as  sensors, 
simulators/stimulators,  radar  devices,  analyzers,  and  other  equipment.  A  Department  of 
Defense  (DoD)  test  or  training  exercise  will  integrate  a  mix  of  these  resources  with  actual 
weapon  systems  and  their  operators.  A  typical  exercise  may  include  firing  a  test  torpedo  at  a 
drone  undersea  vehicle  and  tracking  the  torpedo  through  the  use  of  an  in-water  hydrophone 
array.  Cohen,  Dunn,  and  Soule  provide  a  full  description  of  the  core  asset  base  and  the 
process  NUWC  used  to  build  the  core  asset  base,  field  initial  products,  and  establish 
management  controls  [Cohen  02]. 


1.1  Measurement  Programs 

A  measurement  program  identifies  and  defines  software  measures  to  support  an 
organization’s  business  goals  [Park  96].  These  measures  provide  insights  into  the 
management  issues  that  the  organization  considers  most  critical  to  its  success.  In  establishing 
a  measurement  program,  the  organization  selects  measures  traceable  to  its  business  goals. 
This  “goal-driven”  approach  assures  management  that  data  collection  activities  stay  focused 
on  the  organization’s  objectives. 


1  NUWC  is  the  U.S.  Navy’s  full  spectrum  research,  development,  test,  evaluation,  engineering,  and 
fleet  support  center  for  submarines,  autonomous  underwater  systems,  and  offensive  and  defensive 
weapon  systems  associated  with  undersea  warfare.  The  team  participating  in  this  case  study  is 
from  the  Ranges,  Engineering,  and  Analysis  Department  at  Division  Newport  and  focuses  on  the 
design,  development,  deployment,  and  maintenance  of  undersea  range  systems.  References  to 
NUWC  in  this  case  study  refer  to  this  specific  team,  not  the  entire  organization. 
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In  goal-driven  measurement,  the  primary  question  is  not  “What  metrics  should  I  use?”  but 
rather  “What  do  I  want  to  know  or  learn?”  [Rombach  89].  Because  the  answers  depend  on  the 
organization’s  goals,  no  fixed  set  of  measures  is  universally  appropriate.  Instead  of 
attempting  to  develop  generic,  all-purpose  lists  of  questionably  useful  measures,  teams  and 
individuals  identify  and  define  measures  that  provide  insights  into  their  own  management 
issues.  The  process  shown  in  Figure  1  helps  an  organization  identify  the  data  it  needs  to  track 
to  answer  the  primary  question.  Steps  1-4  of  the  process  direct  the  organization  to  identify  its 
business  goals.  Steps  5,  6,  and  8  produce  measurement  goals  that  are  necessary  to  identify 
what  information  is  most  useful  to  managers  (indicators)  and  to  determine  success  in  meeting 
the  business  goals.  In  Steps  7  and  9,  the  organization  identifies  the  data  elements  (measures) 
it  must  collect,  how/when  to  collect  them,  and  how  to  report  them.  A  measurement  planning 
step  (Step  10)  guides  the  creation  of  a  written  plan  that  the  organization  and  projects  will  use 
to  set  up  the  measurement  program.  Details  pertaining  to  each  step  will  be  given  in 
subsequent  sections  of  this  report. 


Figure  1:  Goal- Driven  Software  Measurement 


1.2  The  NUWC  Product  Line  Activity 

NUWC  has  an  established  product  line  effort  and  has  moved  through  identifiable  phases  in 
its  development  and  use  of  RangeWare  core  assets  to  field  range  systems  [Cohen  99].  The 
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core  asset  development  activity  involved  the  creation  of  the  initial  core  asset  base  and  its  use 
in  the  development  of  a  small  number  of  products.  After  the  pilot  applications  of  Range  Ware, 
NUWC  used  the  architecture  and  components  to  develop  new  product  line  systems,  improve 
the  core  assets,  and  refine  processes  such  as  configuration  management  for  core  asset  and 
product  development.  The  concentration  on  core  asset  sustainment  and  product  development 
characterizes  a  second  phase  in  product  line  institutionalization.  With  new  projects  currently 
underway,  NUWC  will  expand  the  coverage  of  the  core  asset  base  and  institutionalize 
practices  from  the  Framework  for  Software  Product  Line  Practice SM  developed  by  the 
Carnegie  Mellon®  Software  Engineering  Institute  (SEI)  [Clements  04].  NUWC  is  also 
looking  for  the  most  effective  and  efficient  ways  to  support  its  customers  as  it  acquires 
systems  from  NUWC’s  development  group. 

The  rest  of  this  report  examines  NUWC  ’s  application  of  the  “Data  Collection,  Metrics,  and 
Tracking”  practice  area  from  the  Framework  for  Software  Product  Line  Practice.  To  put 
their  measurement  program  in  place,  NUWC  representatives  participated  in  a  Goal-Driven 
Measurement  Workshop.  During  that  workshop,  they  developed  a  preliminary  description  of 
goals,  subgoals,  and  information  requirements.  With  assistance  from  the  SEI,  NUWC  refined 
these  preliminary  results  into  a  single  business  goal  statement,  developed  formalized 
subgoals,  and  identified  indicators.  NUWC  also  established  a  measurement  group  with 
representatives  from  four  range  system  projects.  As  of  the  publication  date  of  this  report,  four 
projects  have  embraced  the  measurement  program  and  are  collecting  data  and  reporting 
results. 


1.3  Report  Organization  and  Related  Work 

Section  2  of  this  report  describes  the  formation  of  the  product  line  measurement  group  and 
the  projects  that  are  currently  participating  in  the  measurement  program.  It  also  describes 
how  the  measurement  group  applied  the  10  steps  of  the  Goal-Driven  Measurement  process  in 
the  following  terms: 

•  establishing  business  goals  and  developing  questions  for  the  measurement  program 

•  identifying  those  indicators  that  the  measurement  program  will  provide  to  managers  to 
help  them  gauge  the  effectiveness  of  the  product  line  approach 

•  identifying  the  data  elements  and  sources  of  information  to  produce  the  indicators 

Section  3  of  this  report  includes  the  early  results  of  the  NUWC  measurement  program. 
Section  4  lists  the  next  steps  to  improving  the  overall  measurement  process. 

While  this  report  deals  with  a  measurement  program  at  a  single  organization,  a  companion 
report  titled  Measures  for  Software  Product  Lines  discusses,  in  general,  the  measurements 

SM  Framework  for  Software  Product  Line  Practice  is  a  service  mark  of  Carnegie  Mellon  University. 

®  Carnegie  Mellon  is  registered  in  the  U.S.  Trademark  and  Patent  Office  by  Carnegie  Mellon 
University. 
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that  a  product  line  organization  is  likely  to  collect  and  use  [Zubrow  03].  That  report  presents 
indicators  and  measures  in  terms  of  product  line  objectives  (performance,  compliance,  and 
effectiveness)  for  management  roles  within  the  product  line.  An  organization,  such  as 
NUWC,  would  select  those  indicators  and  measures  that  allow  it  to  gauge  the  level  at  which 
it  is  meeting  its  business  goals  or  product  line  objectives. 
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2  The  Software  Measurement  Program  at  NUWC 


In  establishing  its  measurement  program,  NUWC  pursued  an  Initiation  Phase  followed  by  a 
Performance  Phase  [Clements  04].  In  the  Initiation  Phase,  NUWC  did  the  following: 

1.  Designated  the  organizational  goals  to  be  tracked 

2.  Defined  the  indicators  (metrics)  to  be  used  to  track  the  progress  toward  NUWC’s  goals 

3.  Identified  the  data  to  collect  to  satisfy  the  indicators 

4.  Identified  the  actions  needed  to  implement  the  measurement  program  including  how  and 
when  to  collect  the  data  and  who  will  collect  the  data  and  report  the  results 

5.  Prepared  a  plan  for  the  measurement  program 

As  of  the  publication  date  of  this  report,  NUWC  was  just  beginning  the  Performance  Phase  of 
its  measurement  program.  In  carrying  out  the  plan  for  the  program,  NUWC  will  pursue  the 
following  steps: 

1 .  Collect  the  specified  data. 

2.  Analyze  the  collected  data  and  translate  it  into  the  indicators. 

3.  Determine  the  management  or  engineering  actions  needed  to  remedy  any  discovered 
issues. 

4.  Verify  whether  those  actions  were  appropriate  for  addressing  those  issues. 

The  reminder  of  this  section  describes  the  approach  NUWC  took  in  establishing  its 
measurement  program. 


2.1  Software  Measurement  Team 

Cohen,  Dunn,  and  Soule  describe  the  early  stages  in  NUWC  ’s  institutionalization  of  its  range 
systems  product  line  [Cohen  02].  The  organization  successfully  fielded  a  small  number  of 
systems  to  its  customers.  It  collected  some  rough  measures  of  the  products  it  delivered 
(including  the  size  of  the  applications)  and  used  that  data  to  form  a  high-level  business  case. 
The  business  case  showed  that  the  use  of  Range  Ware  in  fielding  range  systems  could  reduce 
the  overall  costs  by  30%  or  more.  However,  the  data  used  to  make  the  case  was  collected 
largely  after  the  fact  and  could  only  validate  the  intuitive  notion  that  product  line 
development  lowers  development  costs.  The  lower  development  costs  were  in  the  form  of 
cost  avoidance  when  compared  to  more  traditional  methods.  The  reduction  in  those  costs  did 
not  result  in  a  refund  to  any  single  sponsor,  but  rather  in  an  ability  to  get  work  done  under 
tighter  budget  constraints. 
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To  obtain  more  precise  data  that  justifies  the  continued  refinement  of  the  product  line 
approach,  NUWC  instituted  a  measurement  program.  Under  sponsorship  of  the  SETs 
Acquisition  Support  and  Product  Line  Systems  Programs,  the  SEI  delivered  training  and 
support  in  goal-driven  measurement.  NUWC  identified  four  projects  as  the  initial  set  of 
participants  in  its  measurement  program: 

1.  Offboard  Advanced  Systems  Stimulus  (OASYS) 

(Two  separate  development  efforts  have  stemmed  from  this  project.) 

2.  Range  Ware  Improvement  (RWI) 

(a  project  separate  from  the  RangeWare  core  asset  base) 

3.  Fixed  Naval  Gunfire  Support  System  (FNGSS) 

4.  Atlantic  Underwater  Test  and  Evaluation  Center  (AUTEC)  Shallow  Water  Mine  Range 
(ASWMR) 

The  measurement  team  also  used  data  from  a  project  nearing  completion,  the  East  Coast 
Shallow  Water  Test  Range,  as  an  example  of  the  types  of  data  to  collect.  Each  project  listed 
the  above  plans  to  use  the  RangeWare  core  assets.  They  may  also  use  other  legacy  core 
assets  supported  by  NUWC. 

NUWC  established  a  measurement  team  with  representation  from  each  of  these  projects.  The 
team  has  reviewed  the  business  goals,  indicators,  and  data  collection  requirements  to  assess 
their  compatibility  with  individual  project  goals.  Members  of  the  team  are  also  looking  at 
tool  support  for  the  measurement  program.  NUWC  currently  uses  TRACKWeb  for  the 
configuration  management  and  bug  tracking  of  RangeWare  and  other  core  asset  bases. 
Projects  under  development  or  maintenance  also  use  TRACKWeb.  In  the  future,  this  tool 
may  support  data  collection  for  the  measurement  program. 


2.2  Goals  and  Questions  for  a  Measurement  Program 

NUWC  followed  Steps  1-4  in  the  Goal-Driven  Measurement  process  to  establish  goals  and 
questions  for  the  measurement  program.  Corporately,  NUWC’s  goal  is  “working  together  to 
deliver  the  best  solutions — quickly.”  In  this  context,  “working  together”  means  working  with 
the  fleet,  industry,  government  laboratories,  and  academia  to  harness  the  solution  with  the 
best  value  for  the  Navy.  The  measurement  team  members  felt  the  primary  business  goal  for 
the  range  software  product  line  was  to  support  their  corporate  charter  in  their  business  area. 

To  do  so,  they  would  maintain  a  leadership  role  in  the  development  of  software  systems  for 
(primarily  undersea)  ranges  and  related  range  systems.  Their  charter  articulated  this  goal  as 
follows: 


Overall  business  goal:  “Deliver  the  best-value,  high-quality  range  software  systems 
for  our  customers — quickly.” 
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A  key  step  in  meeting  this  goal  is  to  accurately  predict  the  cost  and  schedule  of  projects  that 
deliver  these  systems.  Given  the  use  of  Range  Ware,  NUWC  hopes  to  achieve  very  high 
reliability  in  making  these  predictions.  For  these  reasons,  the  measurement  team  set  the 
following  subgoal: 

Business  subgoal  1:  “Improve  the  accuracy  of  the  cost  estimation  process.” 

A  second  subgoal  addresses  the  issue  of  product  line  effectiveness: 

Business  subgoal  2:  “Measure  the  effectiveness  of  the  product  line  approach  in  terms 
of  cost  savings  over  the  non-product-line  approach.” 

To  be  viewed  as  a  provider  of  the  “best- value”  solutions  and  as  an  “honest  broker”  among 
industry,  laboratory,  and  academic  suppliers,  NUWC  must  meet  or  exceed  the  expectations  of 
range  customers.  The  “honest  broker”  role  of  a  government  laboratory  is  distinctly  different 
from  private  industry.  NUWC  must  offer/integrate  the  best-value  solutions  to  customer 
requirements  as  opposed  to  selling  a  solution  that  will  create  the  most  profit.  This  led  to  the 
third  subgoal: 

Business  Subgoal  3:  “Ensure  that  all  products  meet  customers’  requirements  and 
expectations.” 

This  subgoal  acknowledges  that  additional  effort  is  required  in  documenting  and 
communicating  requirements  and  expectations.  This  requires  a  close  working  relationship 
with  both  customers  and  sponsors  to  understand  the  customers’  needs,  the  operating 
environment,  and  the  sponsors’  resources.  The  extra  effort  ensures  that  the  requirements  are 
being  met  and  the  developers  are  not  merely  responding  to  perceived  shortfalls  in  a  system 
(i.e.,  solutions  meet  a  known  requirement  versus  requirement  creep). 

Business  subgoals  should  not  exist  for  their  own  purposes;  they  should  address  some  real 
need  within  the  product  line  approach.  NUWC  refined  each  of  its  subgoals  (Steps  3  and  4  of 
Figure  1)  to  identify  meaningful  entities,  purposes,  perspectives,  and  environments  connected 
to  the  subgoal.  NUWC  considered  several  kinds  of  entities  in  forming  subgoals,  including 
products,  development  artifacts,  organizational  structures,  processes,  and  activities.  To 
establish  a  purpose  for  each  subgoal,  the  motivation  for  collecting  and  reporting  data  had  to 
be  identified.  The  perspective  identified  the  stakeholders  who  were  interested  in  the 
measurement  results.  The  principal  viewpoint  was  that  of  manager  for  the  initial  goal.  The 
customer  will  also  be  a  recipient  of  results.  The  environment  provides  a  context  (processes, 
tools,  and  organization)  for  collecting  and  interpreting  measurement  results. 

For  subgoal  1  (improving  the  accuracy  of  cost  estimation),  the  object  of  interest  was  the 
effort  estimation  process.  While  the  purpose  of  that  subgoal  is  clearly  to  improve  estimation, 
it  also  involves  ordering  and  structuring  work  tasks  from  various  sources  (see  Section  2.3)  in 
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sizes  that  are  measurable  and  trackable.  NUWC  has  set  this  subgoal  to  better  understand  the 
causes  behind  deviations  from  estimates  and  to  relate  those  estimates  to  specific 
characteristics  of  the  task  or  a  class  of  tasks.  Finally,  NUWC  determined  that  using  tools  can 
support  the  tracking  activity,  and  subgoal  1  can  improve  how  existing  tools  are  used  or  help 
determine  where  new  tools  are  required. 

Given  this  understanding  of  the  measurement,  NUWC  began  to  ask  specific  questions  about 
its  product  line  development  process  to  help  achieve  its  stated  goal.  Without  this  refinement, 
only  vaguely  stated  questions  could  be  addressed,  and  they  shed  little  light  on  the  measures 
needed  to  understand  the  fundamental  attributes  of  the  processes  and  products  that  are  part  of 
the  product  line  approach.  Figure  2  provides  a  summary  of  the  refinement  process  for 
subgoal  1  that  leads  to  specific  questions  that  must  be  answered  by  the  measurement 
program. 


Object  of  Interest 

Effort  estimation 

Purpose 

Improve  the  ability  to  make  reliable  estimates  of  effort. 

Perspective 

Analyze  the  causes  of  deviation  from  effort  estimates. 

Environment  and 
Constraints 

Use  TRACKWeb  to  estimate,  track,  and  report  effort 
within  a  project  and  for  comparison  across  projects. 
Interface  with  additional  tools  may  be  required. 

Figure  2:  Refinement  of  Overarching  Goal  into  Subgoal  1 

2.3  Questions  and  Indicators 

To  identify  the  information  most  useful  to  managers,  NUWC  developed  a  set  of  indicators 
following  Steps  5,  6,  and  8  in  Goal-Driven  Software  Measurement.  As  of  the  publication  date 
of  this  report,  NUWC  is  using  the  measurement  program  to  address  subgoal  1  (i.e.,  improve 
the  accuracy  of  the  cost  estimation  process)  in  two  ways: 

1.  tracking  predictions  within  a  project — that  is,  how  well  did  the  project  meet  its 
estimates? 

2.  tracking  predictions  across  projects — that  is,  how  and  why  do  projects  differ  in  their 
abilities  to  make  reliable  estimates? 

This  section  identifies  two  things:  (1)  the  questions  NUWC  asked  developers  in  order  to  find 
out  how  they  measured  their  progress  toward  subgoal  1  and  (2)  the  results  NUWC  hopes  to 
attain. 
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Each  project  was  broken  up  into  a  series  of  tasks.  NUWC  defined  tasks  in  three  ways:  (1)  in 
the  project  work  breakdown  structure  (WBS),  (2)  through  action  items  (AIs),  and  (3)  through 
software  change  proposals  (SCPs).  The  WBS,  AIs,  and  SCPs  had  different  origins: 

•  The  WBS  was  generated  by  NUWC  using  customer  requirements  for  a  given  project. 

•  The  AIs  that  resulted  in  the  generation  of  software  tasks  during  project  reviews  or 
meetings  could  have  originated  from  multiple  sources  (customers,  sponsors,  the 
engineering  team,  etc.),  but  each  AI  had  to  be  considered  trackable  by  the  project. 

•  Regarding  SCPs  submitted  by  a  user  or  developer  and  entered  into  TRACKWeb  to  track 
and  report  on  their  progress,  NUWC’s  configuration  control  board  decided  which  ones 
were  addressed  before  they  became  tasks.  That  board  includes  customer  representatives. 

In  addition,  tasks  differed  widely  in  size  based  on  the  nature  of  the  task  and  how  the  project 
developed  its  WBS.  The  scope  of  most  tasks  was  known  when  defined,  but  others  had 
significant  unknowns  or  included  multiple  subtasks.  Although  the  WBS  lists  individual 
tasks,  the  level  of  detail  for  software-specific  tasks  varied  greatly  from  project  to  project.  In 
some  cases,  software  development  appeared  as  only  a  single  line  in  the  project-level  WBSs 
for  multidisciplinary  projects.  The  software  developers  had  to  break  the  WBS  line  item  down 
to  a  level  that  could  be  adequately  tracked  and  then  estimated  the  effort  required  to  complete 
that  task.  Tasks  were  sometimes  further  divided  into  subtasks  to  obtain  a  maximum  of  a  two- 
week  level  of  effort  for  an  individual  task.  NUWC  asked  the  following  questions  about  these 
tasks: 

•  How  accurately  do  we  estimate  the  effort  to  complete  them? 

•  How  does  the  accuracy  of  estimates  compare  across  tasks  in  a  single  project? 

•  How  does  the  accuracy  compare  across  multiple  projects? 

•  Will  effort  estimates  be  of  comparable  quality  (i.e.,  are  all  projects  equally  good  in 
making  estimates)? 

•  How  will  estimates  vary  based  on  projects  of  different  size?  different  levels  of  reuse? 

•  Will  the  reliability  of  estimates  vary  depending  on  the  source  of  the  task:  WBS,  AI,  and 
SCP  work? 

By  measuring  variance  in  terms  of  cost,  effort,  and  schedule,  NUWC  hoped  to  identify 
deviations  and  their  causes.  The  measurement  program  fed  results  back  to  projects  to 
improve  the  reliability  of  future  estimates. 

For  this  element  of  the  measurement  program,  NUWC  represented  the  results  in  a  series  of 
bar  charts.  Figure  3  shows  the  indicator  for  effort  variance.  NUWC  generated  additional 
charts  for  cost  and  schedule  variance.  Within  the  effort  variance  indicator  and  for  each 
project,  task  variance  was  either  positive  (if  the  actual  effort  exceeded  the  estimate)  or 
negative  (if  the  task  was  completed  in  less  time  then  estimated).  Tasks  were  sometimes 
grouped  by  source  (WBS,  AI,  SCP)  to  address  question  6  above.  NUWC  plans  to  develop  an 
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aggregate  variance  for  a  project  and  then  use  an  indicator  to  compare  variance  across  projects 
to  address  questions  3-5.  NUWC  is  still  developing  a  method  for  aggregating  data  and 
comparing  data  across  projects.  In  addition  to  the  individual  variance  indicators,  NUWC 
plans  to  develop  an  indicator  that  shows  all  three  variances:  effort,  cost,  and  schedule. 
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Figure  3:  Effort  Indicator  for  Improving  the  Accuracy  of  Cost  Estimation 


2.4  Data  Elements  and  Sources 

NUWC  determined  the  specific  data  needs  required  to  answer  the  questions  and  build  the 
indicators  for  each  subgoal.  Focusing  on  specific  data  needs,  NUWC  collected  only  the  data 
actually  needed.  NUWC  connected  each  data  element  to  a  specific  indicator  or  to  multiple 
indicators  if  it  could  serve  multiple  needs. 

Table  1  illustrates  the  relationship  between  the  collected  data  elements  and  the  indicators  for 
subgoal  1.  The  priorities  for  this  subgoal  are  data  elements  8-14,  which  contain  both 
estimated  and  actual  effort,  schedule,  and  cost  data. 
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Table  1:  Data  Elements  that  Are  Collected  for  Subgoal  1 


Indicator 

Data  Elements  Required 

Effort 

Variance 

Schedule 

Variance 

Cost 

Variance 

1  Identifier 

X 

X 

X 

2  Description 

X 

X 

X 

3  Requirement 

X 

X 

X 

4  Baseline  (Effort,  Completion) 

X 

X 

5  Revised  Baseline 

X 

X 

6  Number  of  Revision  (History) 

X 

X 

7  Rationale  for  Revision 

X 

X 

8  Actual  Effort  to  Date 

X 

9  Estimate  Effort  to  Complete 

X 

10  Estimate  Actual  Completion  Date 

X 

11  Revised  Completion  Date 

X 

12  Labor  Resource  Cost 

X 

13  Revised  Costs 

X 

14  Other  Task  Cost 

X 

15  Actual  Figures  (Effort,  Schedule, 
Cost)  at  Completion 

X 

X 

X 

Table  2  illustrates  the  relationship  between  the  collected  data  elements  and  the  indicators  for 
subgoal  1. 
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Table  2:  Data  Elements  that  Are  Computed  for  Subgoal  1 


Indicator 

Data  Elements  Required 

Effort 

Variance 

Schedule 

Variance 

Cost 

Variance 

Compute 

Baseline  Cost  (Labor  and  Other) 

X 

Costs  to  Date 

X 

%  Completed  (Schedule,  Effort) 

X 

X 

%  Actual  Effort  Spent 

X 

X 

Estimate  (Effort,  Cost)  at 

Completion 

X 

X 

Variance  (Effort,  Schedule,  Cost) 

X 

X 

X 

The  identification  of  a  small  number  of  required  data  elements  allays  a  critical  concern  in  the 
data  collection  process.  Given  the  number  of  data  elements  required,  managers  and  engineers 
would  be  concerned  with  the  time  and  dedication  needed  to  accurately  track  information  for 
the  indicators.  NUWC  has  selected  15  data  elements.  Some,  such  as  1-3,  characterize  the 
data  elements.  Others,  such  as  4-7,  represent  estimates  of  cost  or  effort  that  will  be  gathered 
at  the  start  of  a  task  or  project.  After  analysis,  NUWC  found  that  only  one  element — Actual 
Effort  to  Date — would  need  to  be  collected  on  an  ongoing  basis.  The  remainder  of  the  data 
elements  would  need  to  be  gathered  only  once  or,  at  worst,  infrequently  when  a  date  or 
estimate  changed.  Other  measures  required  for  the  indicators  (listed  in  Table  2)  can  be 
computed  for  the  data  elements  themselves  and  therefore  do  not  impose  any  collection  effort 
on  the  engineers. 

While  ongoing  data  collection  requirements  were  minimal,  many  data  elements  were  not 
immediately  available  at  the  initiation  of  the  measurement  program.  In  fact,  the  only 
information  consistently  tracked  about  a  task  is  its  name,  description,  and  baseline  effort. 
Baseline  effort  might  be  updated  over  the  course  of  a  task’s  execution  or  as  a  result  of  an 
activity  associated  with  another  task,  but  a  history  of  these  changes  isn’t  necessarily 
maintained.  The  rationale  for  the  change  might  or  might  not  be  recorded.  NUWC  studied  the 
data  requirements  to  assess  their  availability  and  the  effort  required  to  consistently  track  a 
specific  data  element.  With  only  one  exception,  all  data  could  be  obtained  with  minimal 
additional  effort.  The  chart  also  shows  a  one-to-many  mapping  for  most  of  the  data  elements: 
one  data  element  supports  the  creation  of  multiple  indicators. 

Data  elements,  collected  under  Step  7  of  Figure  1,  come  from  a  variety  of  sources.  Project 
documents  such  as  the  WBS  or  requirements  documents  are  a  source  of  information  about 
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several  of  the  data  elements.  Project  team  members  are  responsible  for  other  data;  they  must 
create  and  record  estimates,  make  revisions  to  those  estimates,  provide  a  rationale  for 
changes,  and  report  the  actual  work  accomplished.  NUWC  does  not  currently  collect  much  of 
this  data  at  the  level  of  granularity  required  to  meet  the  organization’s  new  measurement 
goals.  The  project  personnel  must  be  taught  how  to  collect,  record,  and  report  data  elements. 
Table  3  shows  the  availability  of  each  required  data  element  before  initiation  of  the 
measurement  program.  The  legend  indicates  the  meaning  of  the  symbols  for  availability 
status.  While  none  of  the  required  data  items  were  very  hard  to  capture,  NUWC  recognized 
the  effort  required  to  begin  collecting  data,  essentially  for  the  first  time.  The  third  column  in 
Table  3  shows  the  source  identified  for  providing  information  for  each  data  element. 

Armed  with  the  complete  list  of  required  data  elements,  their  sources,  and  their  applicability 
to  specific  indicators,  NUWC  created  a  data  collection  and  reporting  procedure.  The  next 
section  shows  the  initial  results  of  applying  the  procedure  to  a  single  project. 


Table  3:  Availability  and  Sources  of  Required  Data  Elements 


Data  Elements  Required 

Availability 

Source 

1  Identifier 

+ 

WBS,  SCP,  Al 

2  Description 

+ 

WBS,  SCP,  Al 

3  Requirement 

000 

Requirements  document 

4  Baseline  (Effort,  Completion) 

+ 

WBS  (sometimes),  Al 

5  Revised  Baseline 

00 

Staff 

6  Number  of  Revision  (History) 

00 

Staff  (not  collected  now) 

7  Rationale  for  Revision 

00 

Staff  (not  collected  now) 

8  Actual  Effort  to  Date 

00 

Timesheets,  staff 

9  Estimate  Effort  to  Complete 

00 

Staff 

10  Estimate  Actual  Completion 

Date 

00 

Staff 

11  Revised  Completion  Date 

00 

Staff 

12  Labor  Resource  Cost 

00 

Staff 

13  Revised  Costs 

00 

Staff 

14  Other  Task  Cost 

00 

Staff 

15  Actual  Figures  (Effort, 
Schedule,  Cost)  at  Completion 

00 

Staff 

Baseline  Cost  (Labor  and 

Other) 

0 

Baseline  effort  *  labor  cost 

Costs  to  Date 

0 

Actual  effort  *  labor  cost 
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Table  3:  Availability  and  Sources  of  Required  Data  Elements  (continued) 


Data  Elements  Required 

Availability 

Source 

%  Completed  (Schedule, 

Effort) 

0 

Actual/baseline  *  100 

%  Actual  Effort  Spent 

0 

Actual/to  complete  *  100 

Estimate  (Effort,  Cost)  at 
Completion 

0 

To  complete  -  actual 

Variance  (Effort,  Schedule, 

Cost) 

0 

To  complete/baseline  *  100 
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3  Results 


NUWC  collected  data  for  one  project  and  provided  the  actual  effort  results  to  test  the 
assumptions  of  the  measurement  program.  Table  4  shows  the  results  of  the  “straw  man” 
application  of  the  measurement  process  on  the  East  Coast  Shallow  Water  Test  Range 
(ECSWTR).  The  data  in  this  chart  showed  several  areas,  described  below,  that  needed  to  be 
improved  for  more  effective  collection  and  reporting. 

Tasks  report  a  baseline  across  several  orders  of  magnitude.  The  “Printing  display  of  range 
data”  and  “XyView  doesn’t  support  Lat/Lon”  (to  add  latitude/longitude  to  the  display)  tasks 
are  full  development  efforts  for  the  subsystem.  These  tasks  fall  far  outside  the  two- week  task 
boundary.  Other  tasks  are  small,  one-  or  two-day  efforts. 

•  The  large  number  of  small  tasks  impedes  variance  comparison.  A  few  extra  hours  on  a 
small  task  causes  a  severe  rise  in  variance.  A  shift  of  a  few  hours  on  a  large  task  is  barely 
noticeable.  Separate  baselines  for  comparison  can  be  established  for  large  and  small 
tasks.  For  large  ones,  a  percentage  basis  is  a  reasonable  baseline;  for  small  tasks,  a 
certain  number  of  hours  is. 

•  it  would  be  reasonable  to  establish  the  baseline  in  terms  of  hours. 

•  Although  NUWC  encountered  no  resistance  to  collecting  data,  data  collection  has  not 
become  a  routine  part  of  the  development  process.  Managers  must  often  make  estimates 
to  fill  in  gaps  where  developers  did  not  report  the  actual  results,  and,  for  other  tasks,  the 
effort  was  completed  before  actual  effort  reporting  began. 

•  Estimates  are  currently  imprecise.  NUWC  does  not  have  a  process  in  place  to  account  for 
causes  of  variance.  Clearly,  for  variance  above  a  certain  threshold,  the  factors 
contributing  to  the  inaccuracy  of  the  estimate  should  be  noted.  Among  those  factors 
suggested  are 

-  adding  new  requirements  to  a  task  without  adjusting  the  effort  estimate 

-  attributing  time  to  a  single  task  when  work  was  performed  on  other  tasks  as  well 

-  determining  where  to  account  for  effort  not  directed  to  a  specific  task 

These  results  were  used  to  improve  collection  on  the  four  projects  that  are  now  launching  the 
measurement  effort. 


CMU/SEI-2004-TN-023 


15 


Table  4:  Initial  Results  from  Data  Collection 


Task  or  Software  Change  Request 

Baseline 

Effort 

Actual 
Effort  (est.) 

Effort 

Variance  % 

Notes 

ECSWTR  audio  mixing 

- 

Completed 

ECSWTR  can't  handle  multiple  streams  of  audio. 

Completed 

Network  audio  tool  can  hang  RangeWare. 

- 

Completed 

Hydrophones  are  miscolored  wrt  to  operations. 

- 

Completed 

ECSWTR  UCS  documentation 

0.0 

48.0 

Not  in  original  estimate 

graphical  user  interface  for  node  operations 

40.0 

0.0 

Eliminated 

Evaluate  and  close  finished  software  change  proposals  (SCPs). 

20.0 

20.0 

0.00% 

Ant  scripts  for  builds 

20.0 

40.0 

100.00% 

StripView  objects  always  have  a  history. 

26.4 

34.4 

30.30% 

Code  completed;  needs  review 

XyView  has  trouble  clearing  its  history. 

26.4 

26.4 

0.00% 

XyView  history  doesn't  handle  0  history  points. 

27.2 

27.2 

0.00% 

StripView  always  looks  like  it  receives  data. 

20.0 

116.0 

480.00% 

Code  completed;  needs  review 

XyView  scaling  grid  dialog  needs  improvement. 

8.0 

8.0 

0.00% 

XyView  doesn't  support  Lat/Lon. 

105.0 

105.0 

0.00% 

Lat/Long  grid  on  XY  display 

XyView  ignores  assets  that  don't  update  XY. 

5.6 

8.0 

42.86% 

It  is  supposed  to  do  that! 

XyView  needs  to  display  units. 

9.6 

9.6 

0.00% 

RangePoint  doesn't  document  units. 

8.0 

8.0 

0.00% 

LATR/RDA  applications  don't  close  window  as  expected. 

20.0 

20.0 

0.00% 

Tsunami  starting  from  RW 

20.0 

0.0 

-100.00% 

TextView  should  use  a  DA  instead  of  an  asset. 

20.0 

20.0 

0.00% 

TextView  needs  a  scrollbar. 

40.0 

40.0 

0.00% 

TextView  needs  to  display  time  in  a  reasonable  fashion. 

20.0 

20.0 

0.00% 

StripView/XyView  needs  to  display  units. 

40.0 

48.0 

20.00% 

Code  completed;  needs  review 

Display  classification  of  exercise 

20.0 

20.0 

0.00% 

Printing  display  of  range  data 

400.0 

80.0 

-80.00% 

BitMap  dumps  only 

PrototypeNative  Lat/Long  View 

0.0 

10.0 

Not  in  original  estimate 

FTL  Support 

0.0 

6.0 

Not  in  original  estimate 

As  more  projects  begin  to  collect  and  report  data,  NUWC  expects  to  see  improvements  in  the 
precision  of  its  estimates.  NUWC  will  also  need  to  develop  training  for  team  members  who 
provide  estimates  or  record  the  actual  work  performed,  so  they  maintain  accurate  time  reports. 
NUWC  will  also  make  comparisons  within  and  between  projects  to  determine  reasons  for 
variance  at  the  project  level. 
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4  Next  Steps 


NUWC  will  continue  to  institutionalize  its  measurement  program.  The  next  steps  involve 
expanding  the  coverage  of  measurement  in  both  breadth  (number  of  projects)  and  depth 
(addressing  all  subgoals).  NUWC  will  also  look  at  defining  a  full  production  plan.  These  next 
steps  include  the  following: 

•  In  this  initiation  of  the  measurement  program,  NUWC  has  looked  at  only  one  subgoal 
and  a  few  projects.  As  the  measurement  program  matures,  NUWC  will  complete  the 
characterization  of  its  other  two  subgoals: 

Subgoal  2:  “Ensure  the  effectiveness  of  the  product  line  approach.”  For  this  subgoal, 
which  establishes  the  actual  value  of  the  product  line  effort,  NUWC  will  ask  questions 
such  as 

1 .  Which  core  assets  are  used  most  often? 

2.  What  is  the  degree  of  core  asset  reuse  in  a  project  (i.e.,  what  percentage  of  a 
project  is  derived  from  the  core  assets)?  NUWC’s  goal  is  to  maximize  the 
percentage  of  deployed  systems  using  product  line  core  assets  by  numbers  of 
requirements,  numbers  of  components,  or  lines  of  code. 

3.  At  what  cost,  schedule,  and  effort  are  core  assets  used? 

4.  How  often  are  core  assets  updated? 

5.  At  what  cost,  schedule,  and  effort  do  these  updates  take  place? 

6.  Are  the  updates  planned  in  response  to  bugs  the  result  of  redesigning  other  core 
assets? 

7.  Do  updates  fall  into  specific  categories? 

8.  How  many  new  core  assets  are  developed? 

9.  Are  projects  completed  sooner  and/or  with  less  effort? 

10.  Do  products  have  fewer  defects? 

1 1 .  Can  products  be  changed  faster  when  requirements  change? 

The  measurement  group  has  not  yet  developed  the  specific  indicators  that  will  be  used 
with  this  goal.  However,  at  least  one  indicator  will  show  comparisons  of  the  cost  of 
completing  a  project  with  core  assets  (including  any  allocated  asset  development  costs) 
and  without  them.  It  will  also  consider  applying  measures  other  than  those  at  the  code 
level,  that  is,  looking  across  the  core  assets  including  requirements,  architecture, 
subsystem  design,  testing,  and  planning. 
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Subgoal  3:  “Ensure  that  all  products  meet  customers’  requirements.”  To  address  this 
subgoal,  NUWC  must  define  measures  to  better  understand  the  value  of  the  project  line 
to  its  customers  and  assure  management  that  the  product  line  approach  is  addressing 
range  customers’  and  users’  needs  in  terms  of  requirements  traceability  and  metrics  for 
quality  assurance  (QA)  and  testing.  A  comparison  between  projects  developed  using  core 
assets  versus  projects  using  other  development  approaches  will  require  customer 
feedback  measures  such  as  the  number  of  defect  reports  or  the  lead  time  to  support  range 
user  exercise  requirements. 

•  The  NUWC  software  product  line  activity  remains  a  largely  informal  process.  Project 
teams  use  Range  Ware  core  assets  and  follow  some  common  processes,  but  the  overall 
development  approach  is  not  captured  in  a  production  plan.  In  the  next  round  of  the 
product  line’s  institutionalization,  NUWC  plans  to  look  at  all  of  its  processes,  including 
core  asset  creation  and  sustainment,  product  development  and  maintenance,  and 
management  processes.  The  production  plan  could  also  examine  steps  in  the 
Performance  Phase  of  a  measurement  program  for  product  lines.  (See  Section  2.)  The 
results  of  a  measurement  program  will  help  shape  the  analysis  and  development  of  the 
more  formal  process. 

•  NUWC  will  institute  an  analysis  program  to  make  comparisons  across  projects.  The 
analysis  will  track  improvements  in  overall  estimating,  reductions  in  the  reuse  costs  of 
Range  Ware  or  other  core  assets,  reductions  in  overall  project  costs  due  to  the  use  of  core 
assets,  and  product  effectiveness  as  reported  through  customer  feedback.  Analysis  will 
also  play  a  role  in  NUWC’s  long-term  strategies  for  tasks  such  as  documenting  the 
business  case  for  new  investment  in  the  product  line  and  supporting  a  decision  process 
for  tradeoffs  between  specific  improvements.  For  example,  budget  limitations  may  make 
it  impossible  to  address  all  SCPs.  The  analysis  will  provide  a  more  formal  means  to 
make  judgments  concerning  which  proposals  should  be  approved. 


18 


CMU/SEI-2004-TN-023 


References 


URLs  are  valid  as  of  the  publication  date  of  this  document. 

[Clements  02]  Clements,  Paul  &  Northrop,  Linda.  Software  Product  Lines:  Practices 
and  Patterns.  Boston,  MA:  Addison- Wesley,  2002. 

[Clements  04]  Clements,  Paul  &  Northrop,  Linda.  A  Framework  for  Software  Product 
Line  Practice,  Version  4.2.  http://www.sei.cmu.edu/plp 
/framework. html  (2004). 

[Cohen  99]  Cohen,  Sholom.  Guidelines  for  Developing  a  Product  Line  Concept  of 

Operations  (CMU/SEI-99-TR-008,  ADA367714).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  1999. 
http://www.sei.cmu.edu/publications/documents/99.reports/99tr008 
/99tr008abstract.html 

[Cohen  02]  Cohen,  Sholom;  Dunn,  Ed;  &  Soule,  Albert.  Successful  Product  Line 

Development  and  Sustainment:  A  DoD  Case  Study  (CMU/SEI-2002-TN- 
018,  ADA407788).  Pittsburgh,  PA:  Software  Engineering  Institute, 
Carnegie  Mellon  University,  2002.  http://www.sei.cmu.edu/publications 
/documents/02. reports/02tn0 1 8.html 

[Park  96]  Park,  Robert  E.;  Goethert,  Wolfhart  B.;  &  Florae,  William  A.  Goal- 

Driven  Software  Measurement — A  Guidebook  (CMU/SEI-96-HB-002, 
AD  A3 13946).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie 
Mellon  University,  1996.  http://www.sei.cmu.edu/publications 
/documents/96. reports/96.hb. 002.html 

[Rombach  89]  Rombach,  H.  Dieter  &  Ulery,  Bradford  T.  “Improving  Software 

Maintenance  Through  Measurement.”  Proceedings  of  the  IEEE  77,  4 
(April  1989):  581-595. 

[Zubrow  03]  Zubrow,  Dave  &  Chastek,  Gary.  Measures  for  Software  Product  Lines 

(CMU/SEI-2003-TN-031,  AD A4 18393).  Pittsburgh,  PA:  Software 
Engineering  Institute,  Carnegie  Mellon  University,  2003. 
http://www.sei.cmu.edu/publications/documents/03.reports/03tn031.html 


CMU/SEI-2004-TN-023 


19 


20 


CMU/SEI-2004-TN-023 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching 
existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding 
this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters 
Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of 
Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 


(Leave  Blank)  May  2004 


4.  TITLE  AND  SUBTITLE 

Case  Study:  A  Measurement  Program  for  Product  Lines 


6.  AUTHOR(S) 

Sholom  Cohen,  Dave  Zubrow,  Ed  Dunn 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


9.  sponsoring/monitoring  agency  name(s)  and  address(es) 

HQ  ESC/XPK 
5  Egiin  Street 

Hanscom  AFB,  MA  01731-2116 


Final 


5.  FUNDING  NUMBERS 

F19628-00-C-0003 


PERFORMING  ORGANIZATION 
REPORT  NUMBER 

CMU/SEI-2004-TN-023 


10.  sponsoring/monitoring  agency 

REPORT  NUMBER 


12a  distribution/availability  statement 


12b  distribution  code 


Unclassified/Unlimited,  DTIC,  NTIS 


13.  ABSTRACT  (MAXIMUM  200  WORDS) 

The  Naval  Undersea  Warfare  Center  (NUWC)  Division  Newport  Ranges,  Engineering,  and  Analysis 
Department  applied  product  line  practices  in  the  development  of  systems  for  the  Navy  test  ranges  it  supports. 
To  gauge  the  success  of  its  product  line  effort,  NUWC  must  be  able  to  measure  the  effectiveness  of  the 
product  line  approach  compared  to  more  traditional  development  approaches.  To  do  this,  NUWC  established 
a  software  measurement  team  to  develop  and  monitor  a  measurement  program.  The  team  includes 
representatives  from  programs  using  the  NUWC  core  asset  base,  RangeWare.  Four  projects  are  currently 
contributing  to  the  measurement  program.  This  report  documents  NUWC’s  approach  for  measurement  by 
describing  the  Goal-Driven  Software  Measurement  approach  adopted  by  the  test  range  product  line  effort  and 
by  providing  early  results  of  the  measurement  program. 


14.  SUBJECT  TERMS 

software  measurement,  software  product  lines,  software  metrics, 
product  line  case  study 


16.  PRICE  CODE 


15.  NUMBER  OF  PAGES 

32 


OF  REPORT 

Unclassified 


18.  SECURITY  CLASSIFICATION  OF 

19.  SECURITY  CLASSIFICATION  OF 

THIS  PAGE 

ABSTRACT 

Unclassified 

Unclassified 

NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89)  Prescribed  by  ANSI  Std.  Z39-18  298-102 


